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Ministry of Public and Business Service Delivery 


(GSiC) . 


Government Services Integration Cluster 
Digital Enterprise Engagement Practice (DEEP) 


Engagement Platform 


Conversational Interactive Voice Response (CIVR) 
MVP for MTO Driver and Vehicle Information 


Monitoring and Reporting 


Digital Enterprise Engagement Practice: “Connecting the people and businesses of Ontario with their Government” 
Government Services Integration Cluster: “Enable the best experiences for citizens, business and employees” 
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MVP Reviews and Reporting (GSiC) 


Getting started our focus is - the system is working and not negatively 
impacting the public or agents. Light reporting to follow. 


Calendar Who to report to 
» Feb 2 — Triage reporting for systems — full team -+ Working team 
» Feb 3 - Triage reporting for systems — full team + Owner/Support — Beth 
» Feb 6 — Reporting analysis story — content working team + Directors — Olga, Lisa 
- we will have enough calls over 1000 + ADM’s/Associate and CIO/CCIO 
+ we can do our analysis on this together + Communications will share with DMO 
» Feb 8 - Reporting analysis story - content working team » SWAT 


» Feb 10 - Reporting analysis story - content working team 
+ Feb 13t forward 

+ CMI calendar content meeting — include Kim and Kyle 

- Weekly full team organized by DEEP 

- Document the issues/defects in the list 


+ Measurement — Sharon Stuckless, Rene 
Alverez plus teams 
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MVP QuickSight Reports and Understanding (GSiC) 
How we start - will be a learning experience for how we go forward. 
What we expect to see What actions are we taking 
-+ CIVR calls — 500 to 1000 a day » Track items in Lists — Defect List 
» Calls directed to agents? ~33% redirects a day » Keywords may be added — threshold TBD 
(of the 15% to start) - Redirects — it sounds like you want information 
» Current IVR does not offer agent in a few on — call or look at 
flows — this is a change - Broadcast message updates or if needed 
» How callers many had multiple questions? -+ What can we do for future - backlog 


- Note trends for planning - Do we need to translate this differently 


+ What are they calling about - trend - Checklists example 
» Missed utterances may be higher initially - Daily to make sure it is working providing the 
< Internal users — trying it out service expected — random checks 

- Monthly the links provided are working 


- Yearly translation review & flows 
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Reports and Surveys (GSiC) 


Once we see the usage, behaviours, and caller stories - we may want to develop 
reports for different audiences. 


Ideas for Possible BI Reports Surveys for callers 


» Top 10 missed utterance daily to spot trends + Repeat the Pre-CIVR survey once it is fully 


» Peak time of day trends with averages etc. deployed 


- Have callers listened to the CIVR message and 
got the answer they were looking for? 


» After hours trends 


» Chart of total calls and then transfers to CIVR 


and agents » Do they know more before they call now? 


» Answered, busy, failed or no answer trends or 
weekly or by QID 


» Time in queue to an agent, transfer time 


- Insights on Voice of the Customer 
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(GSiC) 


Accessing QuickSight Reports Quick Access 
. Find the APP 

Two Step Process DA 

1. Setup Account ud Quicksight 

2. Access to view oe 

reports/dashboards will be 


shared 
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CIVR Analytics & Reporting: Amazon QuickSight (GSiC) 
MVP dashboard offers reporting data that can be targeted to specific date ranges. 


é ¢ 


Amazon QuickSight delivers easy to Gd Quicksight 
understand insights through Q 
interactive dashboards allowing for 


Dashboards 


bnrooard T oasroosra 


l 


QnABotMetrics_Eng_Prod3. CIVR Navigational Report 


wi wi 


analysis of data. howeons Bowen 


Reporting data is essential for 
improving CIVR performance, which 
includes learning more about callers 
and resolving technical issues. 


S a 


RACE_REPORTDATAPROD | QnABotMetrics_Fr.P 


Dashboards 


QuickSight reports will assist K 
ServiceOntario and MTO with 8 
iterative improvements to the D&V 
CIVR needed for long term success. 
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Amazon QuickSight: Types of Reporting (GSiC 


1. QnA Bot Metrics 


2. Trace 


3. CIVR Navigation 


+ What: Provides data related to what the CIVR understand the callers are asking, what answers are being provided, what 
words the CIVR is not picking up, and the topics count for select period. 
+ Who: SO CCMI & CMI & MTO Policy/Program Areas 


+ Example: Leverage this report to compare what were the most frequently asked questions that are answered or missed 
utterances that are not. This can provide insight into what specific information customers are looking for from one month to 
another. ACTION: update webpages, and for continuous improvement. 


What: CVIR collects what it believes the callers' actions are while using the CIVR which provides understanding into caller 
behavior. Understanding how easy or difficult the are callers able to navigate the CIVR - if there are many timeouts or max 
invalids that could indicate the CIVR does not understand what the caller is asking, and they are having difficulty. 

+ Who: SO CCMI & CMI 


+ Example: Review the Trace report to see where callers are repeatedly asking CIVR to “repeat” the answers which could be 


an indication that the information presented is not clear or too much information is provided. ACTION: the specific 
script(s) should be reviewed and modified. 


e What: Navigational report which provides a summary of how callers moved through the CIVR - total times a questionID 
was referenced, hang-ups, timeouts, repeats, requests for agents, etc. for each Question ID. 
e Who: SO CCMI/CMI & MTO 


+ Example: ServiceOntario Locations QID was referenced 22 times which had 1 timeouts, 9 repeats and 2 hang-ups. This 
could indicate that the caller was satisfied with the information provided even though there were 9 repeats as the callers 
want to ensure they retain the correct information. Additional with no invalids, timeouts or max repeats/errors this could 
also indicates callers are receiving the information they required and had no further questions. ACTION: — use this report to 
see which QIDs are triggering the most requests for agent transfers and update those QIDs to provide more information. 
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(GSiC) 


View of Metrics, Monitoring & KPIs 


Digital Enterprise Engagement Practice: “Connecting the people and businesses of Ontario with their Government” 
Government Services Integration Cluster: “Enable the best experiences for citizens, business and employees” 
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1. QnA Bot Metrics (GSiC) 


View a dashboard for a time period selected. 
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1. System Monitoring — Questions Answered and Fulfilled 
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(GSiC) 


Learn about what people are asking most — Voice of the Customer. 


Reporting — Starting with similar reports, and word cloud for focus. 

+ Identify the most popular questions that the CIVR believes callers are asking 
within a certain time period 

+ This example indicates callers may have been inquiring about transferring 
ownership 

» Agent utterance was requested and fulfilled 42 within the time period 

+ “| want to pay a driver fine” was the second most popular fulfilled utterance 
indicating most callers are asking about this specific topic or QID 


Usage 
Start Date & End Date 
Shows total number of QIDs referenced 
Questions — Question Count 
Shows number of callers hitting the specific QID including 
main menu, agent and no matches 
Benefit: we can see what the CIVR thinks the callers are 
asking about 
Can sort the most important/popular QIDs 
Answer — Answer Cloud 
Answer scripts played for questions asked 
Provides a count of the most popular answers the CIVR is 
providing 
Fulfilled — Fulfilled Utterances Count 
Utterances that the CIVR was about to fulfill based upon 
the request it understood and provided the response 
Topics — Topics Count 
Can sort most important/popular topics the callers are 
calling about 
Each topic can have many specific QIDs associated with it 


oviechO Ontario VW 


MPBSD 23-071 Record 13 DV-CIVR - Reporting Overview - Launch - 2023-01-25 Page 11 of 21 


1. System Monitoring — Missed Utterances and Word Cloud (GSiC) 


Learn about the what people are asking — Voice of the Customer. 


Reporting 
* Missed utterances can be used to see what CIVR was not able to fulfill and provide an answer for 
* They can also be used to indicate what new questions can be added to the QnABots for the CIVR to recognize 


Missed Utterances Count 
Missed Utterances Count 


utterance Come 
utterance Count K 
i A : > “DV. ‘PV. Renewsticker 
car prices to mention Ov.DL-Apely 
abird 1 feedback 1 DVL DL. DriveTest DV_DL. Drivesticence Renew 
[OV_PV.Permit Replace DV_OL DiglalReminder w 
a one i need to pay 1 gibberish J emp Ly DLO 
a room 1 i have got my face thy 1 Ov. DL Suspensionsfines, DV_PVRegisterVehicle 


a time limit to submit my application to apply... 1 ihust myntrforthecard “T ihave got my ace]? DV_ PV. It n sfer r 


i know i hope it by this 


a well i need to apply for under you for twent... 1 DV “A P ONY. -speciatpermt 
jibrish 1 ter 

aape card 1 S "Dior Ronewatsecsion 
no 2 

sians 1 oh i need information on pocket permit for m... 

allergies russian 1 oh okay ummm 

am | eligible for an a p p 1 oh party but its for existing bilety needs Over time the data will be enriched 

Total 1,194 » Contact handling quantitative statistics will include 
Total 38 


speech analytics capabilities (for clients that use the EP- 
cloud instance) 
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2. Trace Report (GSiC) 


We can trace what people are doing and the time spent in the CIVR. 
Reporting Main Use 
» Which QIDs were presented to the caller 
-+ Potential time spent, invalids, calls going to agents, etc. 
» The number of requests for an agent - could indicate a change required to that QID 


(ad QuickSight 


Trace Reporting x 

* This is a view is available for a Gi a tnd ome 
requested time period 

* Each conversation is assigned 
a code 
Time stamp shows when the 
calls came in (UTC) 

-The conversation is outlined 
by the naming conventions 
for each item 

* Time is noted in number of 
seconds for each named 
convention 


id, and Alirequests 


AllRequests 


OLBEGINDLOS] 0.Globsl_Mandter MainMerdAgentia9s) 


wwcemebocd.serts0501) DV_DL DriveTest_Permanent 


ecumentsapent(24s} 0. Global SmallTalk AreYouThere14s} VOL OriveTest 


DL briveTes 


JV DL DoiveTest_Driversicencl 


katos] BV. OL DrlveTest_Canacle 


M194} Dv. 01.0 


OV DL.Agpty G08) OV DL. Apply. G8] OV. 


View: $00 items V 
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2. Trace Report more 
“Showing and telling” trace report value by example. 


Example: Hang-up 
-Caller reached CIVR (English) and then hung-up 


Contactid DateTime Civrid AllRequests 
86266c01-1bb0-4594-956f-65ca631981ca Nov 21, 2022 Timeout 0_BEGIN_E()[0s] 0_Global_Handler.MainMenu(invalid)[43s} 
bO8fc9e7-01b2-49d1-a2f3-40b1fc13cd46 Nov 21, 2022 2290 


Example: Successful Call 


+ Caller asked about suspension or fines and heard the answer script/information and hung-up 
Caller did not ask for an agent, repeat or any other questions 


€41131d1-cf23-4e83-9f21-f18726669aeb Nov 21, 2022 2289 0_BEGIN_E0[0s] DV_DL SuspensionsFines_info()[40s} 


Example: Request for Agent 
* Caller reached CIVR (English) and then hung-up 


16d07d0-8106-4b52-bd67-fabb764ffd7e Nov 21, 2022 2291 0_BEGIN_E()[0s] O_Global_Handler. MainMenu(Agent)[48s] 


Example: Did not get required information 
* Caller connects with CIVR (French), gets DriveTest_Appointment information three times and 
last attempt timeouts (did not say anything after the answer script was provided) 


5009bdb1-dacc-4075-3c27-5b5f504224c0 Nov22,2022 Timeout 
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(GSiC) 


Definitions 


GovTechON 


Contact ID: The unique identifier 
created from Amazon Connect 


Civrid: The custom ID created 
from Amazon Connect which is 
created every time a call goes 
from on-prem SO D&V IVR to 
CIVR in Amazon 


Timeout: If a call is made directly 
to CIVR it shows timeout, but if a 
call is made to On-prem solution 
first then it provides the CIVRID 
O_BEGIN: The start of all calls 
made into CIVR 

Global Handler.MainMenu: 
Caller utters invalid 


word/phrase, start over, | want 
to start over 


0.BEGIN_FO[0:] DV_OLOriveTest Appointment FO[SE«] DV_DL DriveTest_ Appointment _Flinvalid)(22s] DV_DL OriveTest_ Appointment F(Timecut)[16:) 
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3. CIVR Navigational Report (GSiC) 


We can review a summary of how callers navigated the CIVR. 


Reporting Main Use 
Provides total number of callers and average time spent in the CIVR (displayed in seconds) 
Summary of calls made, total times a questionID (QID) was referenced and specific information on each QID (hang-ups, 
timeouts, repeats, requests for agents, etc.) 
If we want to reduce calls being transferred to agent — use this report to see which QIDs are triggering the most requests for 
agent transfers 


This is a view is pulled for a 
requested time period 
Total number of calls - 12 calls 332 167 165 59.23 
made it to the “Apply_Cost_Info 
QiD” 
The breakdown of 12 calls: 

1 timeout 

1 invalid 

5 repeats 

1 agent request 
Multiple occurrences within a 
single call causes the total and 
breakdown to not equal 


Total Number of callers Number of English cater Number of French callers Average time spent in CIVR (See) 


Note: Columns are similar to the current IVR navigation reports 
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3. CIVR Navigational Report — Hang-up Calls 
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(GSiC) 


Each a topic/queue is identified under question ID — Data helps to understand the story. 


Queue ID Reporting example 

+ Hang-up calls — should we be 
concerned? 
2 calls and 2 hang-ups for 
QID APP address change 
No request to speak to an 


agent, repeat or error out 
The story we can tell here is 
that likely this indicates the 
callers are receiving the 
information they were 
seeking and hung-up 

No concern or action needed 


GovTechON 
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3. CIVR Navigational Report — Max Error (GSiC) 


Question ID data — needs to be understood in the context volumes for action. 


Queue ID Reporting example 

» Max Error — should we be concerned? 
QID was requested 17 times - the CIVR 
believes this for the QID for APP 
Address Change Sanaa 
1 MaxError — is noted which means a QuestiontD pate Toul | Timeouts | braids | epost | siaii | Apem® | sanapas | Daars | [Hecgnp 
caller uttered something that the CIVR ; p ; ; F 3 
could not understand 3 times for QID 
The story we can tell here is that 1 in 17 
callers was not understood by the CIVR 
and they were taken to the agent 


queue. ) OV.PyRenewsticer insurancevertted + 
This is not the primary desired 
experience, however being directed to 
the agent ensures and an answer 

No concern 

Missed utterances can be checked — 
though some volume may be required 
for trends 


J DV_PV.ReplaceLoststolendamagePt 2 2 7 
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KPI Measures (GSiC) 


Success is about more than delivering a project on time and on budget — it is about 
meeting our callers need for service, both to self-serve and to reach an agent. 


[Measure | Deseripton asctne [aret | 


Baseline survey 
* caller is asked if they are willing to participate in a survey 


Customer + if they agree they are able to confirm their phone number 72.40% 80% 
satisfaction * The system places an outbound call to the caller to conduct the survey j 
(overall) CIVR — Relaunch Survey 


* Callback to callers after the call (80% Standard) - approvals required 
* CIVR survey tool to connect to be developed 
Speed to need Baseline - time for callers to reach first topic in IVR 


(Time to first CIVR - time for callers to reach first topic 112 sec 100.8 sec 
topic) * compare against baseline to measure improvement. (-10%) 

Baseline - time for callers to reach MMR in IVR by pressing 0 
Speed to need ciyp - time for callers to reach MMR 42 sec S7 aea 


(Time to MMR) . compare against baseline to measure improvement (-10%) 


Baseline - measured in IVR (hang-up /total callers) 
% Self serve. Any "hang-up" in a topic will be considered a self-serve 37% 33.3 % 
(% hang ups) IVR. Aspirational it would increase, but success is maintaining (-10%) 
Request Agent Baseline - number of callers that press 0 from top menu/total calls 7 A 
(%top menu) CIVR - number of callers that ask to speak to an agent immediately (-10%) 10% 9% 
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(GSiC) 


QUESTIONS 


Ontario WY 


MPBSD 23-071 Record 13 DV-CIVR - Reporting Overview - Launch - 2023-01-25 Page 19 of 21 


Amazon QuickSight & EP iceReports (GSiC) 


The caller experience and insights from reports. 


AWS CIVR Reporting Enterprise Platform On-Prem iceReports 
* Callers experience with the CIVR Solution * Callers experience after being transferred to an 
before agent transfer agent - survey 
* Insights for business: * Insights on users/agents experience: 
* Caller trends, popular requests and caller * Number of calls handled by agents, calls 
preferences transferred, etc. 
* Insights for operations: * Insights on queue activity: 
* Identify potential problems that could * Number of calls that were in queue (avg. wait 
impact customer experience times, abandon time, on call, etc.) — need 


, . volume for impact. 
e Review performance metrics 
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Amazon QuickSight & ECCS iceReports Flow 


The caller experience and flow for reports. 


Enterprise Platform AWS 
Conversational IVR 


AWS QuickSight Reporting 


Main menu 


EON 
(oma i “Front end message you aling about. 


driver s license?” 


Callers question(s) 


Lets get started. Ask a question or say what 


For example, you can say "How do I renew my | 


Enterprise Platform 
On-Prem ice Soultion 


Sends calito ECCS 


>| answered, but 
wants confirmation 


20 Presentation Name 


| ECCS iceReports 


[Connects 
| Caller placed into | | 
wth agent using sal Clients questions) 
4—>} queve for an net H f > lends 
svalableagent, | | ECCS om-premicE answered a 
platform 
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Defect List (GSiC) 


100% of the time we are tracking issues and defects to understand and plan. 


Transparent Tracking 

* Daily planner was used for chatbot — we have decided to try a new route 
* This will include actual defects and who it is assigned to 

* The features can grow based on need - agile 


iii Microsoft Lists 


kie E Edt in grid view 1È Shao v $ Export tograte All tems 
as Conversational IVR 
Defect List * © 
g 1D Utterance said (what the tester said ver. Expected QID R. ‘Actual Response Priority Status Comments Add column, 
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